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(57) Abstract: A system for rate control during transcoding and encoding of digital video data in a multi -program transmission 
environment accommodates changes in the GOP structure of any of the programs, e.g., due to a commercial insertion or the like. A 
hierarchical bit allocation scheme is used, where bits are allocated on a super group of pictures (GOP) level (106), i.e., a grouping of 
multiple GOPs, a super frame level ( 104), and an individual frame level (102). The bit allocation for the super GOP and each picture 
is adjusted when a change in the structure of a group of pictures (GOP) is detected for any of the pre -compressed programs. The 
super GOP length may also be adjusted if required to maintain an integer number of GOPs therein. A change in the GOP structure 
is detected based on a change in the GOP length or sub_GOP length (distance between P pictures). For pre -compressed data, initial 
GOP lengths and sub_GOP lengths can be assumed and adjusted as required based on actual measurements. For a current I or P 
picture, the sub_GOP length is calculated based on temporal references of the current picture and a previous I or P picture. For a 
current B picture, the same calculation is used, plus one picture. 
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STATISTICAL MULTIPLEXER AND REMULTIP LEXER THAT 
ACCOMMODATES CHANGES IN STRUCTURE OF GROUP OF PICTURES 

BACKGROUND OF THE INVENTION 

The present invention relates to rate control 
5 during transcoding and encoding of digital video 

programs in a multi -program transmission environment, 
where several programs are multiplexed and transmitted 
over a single communication channel. 

Commonly, it is necessary to adjust a bit rate of 

10 digital video programs that are provided, e.g., to 

subscriber terminals in a cable television network or 
the like. For example, a first group of signals may be 
received at a headend via a satellite transmission. The 
headend operator may desire to forward selected programs 

15 to the subscribers while adding programs (e.g., 

commercials or other content) from a local source, such 
as storage media or a local live feed. Additionally, it 
is often necessary to provide the programs within an 
overall available channel bandwidth. It may also be 

2 0 desired to change the relative quality level of a 
program by allocating more or fewer bits. 

Accordingly, the statistical multiplexer (stat 
mux) , or encoder, which includes a number of encoders 
for encoding uncompressed digital video signals at a 

25 specified bit rate, has been developed. The statistical 
remultiplexer (stat remux) , or transcoder, which handles 
pre-compressed video bit streams by re -compressing them 
at a specified bit rate, has also been developed. 
Moreover, functions of a stat mux and remux may be 
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combined when it is desired to transcode pre-compressed 
data while also coding uncompressed data for transport 
in a common output bitstream. Uncompressed programs are 
coded for the first time, while compressed programs are 
5 re-encoded, typically at a different bit rate. 

For MPEG applications, a stat mux/remux must 
accommodate three different picture types (I, P and B) , 
which usually require quite different numbers of bits 
because of the different nature of their temporal 

10 processing. 

Bit allocation strategies should take the picture 
types into consideration. For a given bit budget, this 
implies a requirement of a priori knowledge of the 
picture organizations (GOP) of the programs. The 

15 requirement is not a problem for encoders because 

encoders can plan ahead for types and arrangement of 
pictures that it will output. However, such a priori 
knowledge is typically not available for transcoders, 
which deal with pre-compressed video bit streams. 

20 Accordingly, it would be desirable to provide a 

stat mux/remux system that removes the assumption that 
the picture organizations are available for all the 
programs . 

The system should provide a novel adaptive bit 
25 allocation strategy for a stat mux/remux system that 

requires no a priori (beforehand) knowledge of program 
picture organization. The system should be able to 
address any changes in program picture organizations, 
including a change in the GOP length and/or the sub-GOP 
30 length (the distance between two P-pictures in either 
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encoding or display order) . 

The system should include transcoders for 
processing pre-compressed video bit streams, or both 
encoders and transcoders, for additionally handling 
5 uncompressed digital video signals. 

Moreover, the system should accommodate changes in 
the group of pictures (GOP) structure of pre-compressed 
bitstreams, for example, due to switching channels, 
commercial insertion, changes in the program content 

10 (e.g., due to a transition from a movie to a news 

announcement or a sports event) , and the like. The 
system should handle structure changes that occur at a 
GOP boundary or within a GOP. 

The system should accommodate changes in a GOP 

15 length and/or a sub_G0P length. 

The system should overcome difficulties in 
integrating pre-compressed program bit streams (that 
include, e.g., video materials such as films and the 
like that are pre-compressed and stored for subsequent 

20 transmission) with uncompressed digital video sequences. 

The system should not require a priori (beforehand) 
knowledge of the picture organization (GOP structure) of 
programs (e.g., the GOP length, and arrangement of 
different picture types in the GOP) . The system should 

25 avoid a processing delay of about one GOP which would 
otherwise be incurred to extract the complete GOP 
structure information from a pre-compressed bit stream. 
The system should also avoid the need to store the data 
corresponding to the GOP, thereby reducing the memory 

3 0 size required for transcoding. 
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The present invention provides a system having the 
above and other advantages. 
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SUMMARY OF THE INVENTION 

The present invention relates to rate control 
during transcoding and coding of digital video data in a 
multi -program transmission environment . 
5 The invention accounts for changes in the structure 

of a GOP, e.g., due to a commercial insertion or the 
like, to adjust the bit allocation for each picture 
accordingly. A change in the GOP length (increase or 
decrease) , and/ or a sub_GOP length can be accommodated. 

10 For example, a change in picture sequence from 

IBBPBBPBB... to IBPBPBPB... results in the sub_GOP changing 
from three to two. 

The sub_G0P length contains information about the 
specific picture types and their arrangement in a GOP. 

15 Hence, even when the GOP length remains the same during 
a change in the input video source, we may have a 
different arrangement of picture types if the sub_GOP 
changes . 

Moreover, the invention operates with a 
20 hierarchical bit allocation scheme, where bits are 

allocated on a super GOP, super frame, and individual 
frame levels. 

The change in the GOP structure can occur anywhere, 
e.g. , within a GOP or super GOP, or at a boundary of a 
25 GOP or super GOP. Once the GOP of a program changes, 
the length of the super GOP is re -calculated, and the 
allocation of bits to the super GOP is adjusted 
accordingly. 

If there is a switch during the middle of an old 
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GOP to another program of a new GOP, this is treated as 
a transition from the old GOP to an old incomplete GOP, 
and then to the new GOP. 

A particular method in accordance with the 
5 invention is provided for coding a plurality L of video 
programs. The method includes the step of allocating 
bits for coding the video programs according to a 
hierarchical scheme that includes at least: (a) a super 
group of pictures (GOP) level, wherein a super GOP 

10 comprises at least one GOP from each of the video 
programs, and (b) an individual frame level. The 
allocating step is responsive to respective GOP 
structures of the video programs. The respective GOP 
structures " of the video programs are monitored to detect 

15 any changes thereof. When a change in the GOP structure 
of at least one of the video programs is detected, the 
allocation of bits for coding the video programs in the 
super GOP is adjusted according to the change. 

Accordingly, the invention can optimize the 

2 0 allocation of bits since a rate control processor 

maintains updated knowledge of each picture type and the 
arrangement of pictures for the different programs to 
the maximum extent possible. 

Corresponding apparatuses are also presented. 
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BRIEF DESCRIPTION OF THE DRAWINGS 

FIG. 1 illustrates a stat mux/remux with both 
encoders and transcoders, and a joint rate control 
engine, in accordance with the present invention. 
5 FIG. 2 illustrates a super group-of -pictures (GOP) 

construct in accordance with the present invention. 

FIG. 3 illustrates a super frame construct in 
accordance with the present invention. 

FIG. 4 illustrates a decoder for receiving data 
10 from the stat mux/remux of FIG. 1 in accordance with the 
present invention . 

FIG. 5 illustrates a video program with a change in 
the sub__GOP length, and subsequent adjustment of the 
super GOP length, in accordance with the present 
15 invention. 

FIG. 6 illustrates a video program with an increase 
in the GOP length in accordance with the present 
invention. 

FIG. 7 illustrates a video program with a decrease 
2 0 in the GOP length in accordance with the present 
invention. 
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DETAILED DESCRIPTION OF THE INVENTION 

The present invention relates to rate control 
during transcoding and coding of digital video data in a 
multi -program transmission environment 
5 1 . INTRODUCTION 

With recent advances in digital video compression 
and digital transmission, it is possible to deliver 
several video programs in the same bandwidth presently- 
occupied by a single analog TV channel. To be squeezed 

10 into a fixed-rate channel, the video programs have to 
share the channel capacity. Specifically, raw pixel 
data is compressed, and compressed bits are transcoded. 
Encoders and transcoders can be thought of as rate- 
conversion engines. An encoder compresses a digital 

15 video sequence into a bit stream at a much lower rate 
while a transcoder converts a pre-compressed video bit 
stream into another bit stream at a new (low) rate. The 
aggregate rate of these compressed bit streams, however, 
has to be equal to, or less than, the channel rate. 

2 0 This can be achieved by controlling either each 

individual rate (independent coding) or the aggregate 
rate (joint coding) . 

In independent coding, rate control can only be 
performed across the time and spatial dimensions of a 

25 single program. In joint coding, control is extended to 
the program dimension (e.g., multiple programs). This 
implies more freedom in allocating the channel capacity 
among programs and, therefore, more control of picture 
quality among multiple programs as well as within a 
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program. 

A system that is able to put multiple video 
programs into a fixed-rate channel is called a: 

1. stat mux if the inputs are all uncompressed 
5 digital video signals, 

2. stat remux if the inputs are all pre-compressed 
video bit streams, or 

3 . stat mux/remux if the inputs are both 
uncompressed digital video signals and pre-compressed 

10 video bit streams. 

A stat mux has encoders, while a stat remux only . 
needs transcoders, and a stat mux/remux has both 
encoders and transcoders . 

FIG. 1 illustrates a stat mux/remux system in 

15 accordance with the present invention. The inputs to 
the stat mux/remux system 100 can include uncompressed 
digital video sequences and/or pre-compressed bit 
streams. The uncompressed digital video sequences, 
e.g., programs 1 and 2 are encoded by encoders 12 0 and 

20 130, respectively, e.g., which use the MPEG video coding 

standard. The pre-compressed programs (e.g., 
bitstreams) L-l and L are processed, respectively, by 
transcoders 140 and 15 0, GOP structure monitors 142 and 
152, and optional bit rate functions 144 and 154. 

25 Any known transcoder configuration may be used. 

The example GOP structure monitors 142 and 152 
operate as will be discussed in greater detail below to 
provide the GOP lengths and N L , and sub GOP lengths 

M h _ x and M L , of the respective (L-l)th and Lth programs 

3 0 to the rate control processor 110, which adjusts the bit 
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allocation to the different programs (via the target bit 
rate T) in response thereto. 

The example bit rate functions 144, 154 are 
optional and are used to monitor the bit rate of the 
5 pre-compressed input data frames. As discussed further 
in connection with equation 14, the rate control 
processor may reduce the target bit rate for a 
transcoder so that it does not exceed the measured bit 
rate . 

10 Note that the invention is operative when only pre- 

compressed bitstreams are present. Uncompressed program 
data is accommodated, but is not required. Moreover, 
any number of pre-compressed bitstreams may be 
accommodated, although only two are shown in FIG. 1 for 

15 simplicity. 

The transcoded data and encoded data (when present) 
are provided to a MUX 160 and buffer 170 prior to being 
transmitted on a channel, typically to a decoder 
population. The channel may be part of a broadband 

2 0 communication network, such as a cable or satellite 

television network, for example. 

The pre-compressed program data may be provided 
from any source, such as a storage medium (e.g., 
magnetic tape or compact disc) , or from a satellite 
25 transmission or stat mux, for example. Pre-compressed 
data, such as for commercial insertion or insertion of 
local news or other programming, may be provided locally 
for use in conjunction with pre-compressed data that is 
received from a remote source. 

3 0 The buffer 170 sends a fullness level signal to a 
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rate control processor 110, which may adjust the target 
bit allocations accordingly to avoid a buffer overflow 
or underflow. 

A user interface 108 may communicate with the rate 
5 control processor 110, for example, to set a GOP 
structure for the encoders 120, 130, to designate 
certain encoders or transcoders as having a higher 
priority, and so forth. 

The stat mux/remux 100 provides a relatively 

10 uniform picture quality (or other designated quality 
level when different program priorities are assigned) 
within a program, and if necessary, among programs. To 
achieve this goal, channel capacity is dynamically 
distributed among programs according to a program 

15 priority as well as a frame level program complexity 
measure . 

Each MPEG encoder 120, 130 or transcoder, 140, 150 
receives a target number of bits, T lf T 2 , and T L , 

respectively, from a rate control processor 110 at each 

20 frame. The rate control processor 110 includes a super 
GOP level processing function 106, a super frame level 
processing function 104, a frame level processing 
function 102, and a complexity processor 105. These 
processing functions may share common hardware such as 

25 memory and processing chips, but are shown individually 
for simplicity . 

The target number of bits for each frame of a 
program is met by adjusting the quantization parameter 
in the MPEG encoder or transcoder. The resulting number 

3 0 of compressed bits, R, as well as the average 
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quantization parameter, Q, used for each frame are then 
sent to the rate control processor 110 as feedback data. 
Specifically, the encoder 120, encoder 130, transcoder 
14 0 and transcoder 150 produce R x , R 2/ R L _ X and R L bits, 
5 respectively, using quantization parameters Q 17 Q 2 , Q L _ 1 
and Q L , respectively. The complexity processor 105 
calculates corresponding complexity values C using R and 
Q for each program. The rate control processor 110 then 
determines a new target number of bits for each new 

10 program frame or picture based on the program complexity 
at the frame level. 

The rate control processor 110 is also responsive 
to the GOP structure data from the transcoders 14 0, 15 0. 
Ideally, a stat mux/remux system distributes the 

15 channel capacity over the input programs according to 
the programs' relative complexity measures. That is, 
more complex programs are assigned more bits, and less 
complex programs are assigned fewer bits. For a given 
rate, this also implies that the rate assigned for a 

2 0 program depends on the complexity measures of the 
program and the other programs. Since the program 
complexity may vary with time, the programs 1 relative 
complexity measures vary as well. The distribution of 
channel capacity, or bit allocation, over programs 

25 therefore has to be a time-varying function (e.g., on a 
frame -by- frame basis) . 

Furthermore, MPEG (ISO/IEC (MPEG-2) , "Generic 
coding of moving pictures and associated audio" , March 
1994) defines three picture types in terms of temporal 

30 processing (I, P and B) , and the organization of the 
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three pictures (GOP) can be very flexible. Due to the 
different nature of their temporal processing, the three 
pictures may require very different numbers of bits. 
Hence, to wisely use the bit budget over the input 
5 programs, the program picture organizations should also 
be taken into consideration. However, for a transcoder, 
the program picture organization is embedded in the pre- 
compressed bit stream and is therefore not available as 
the bit stream is being received. While a transcoder 

10 can learn the program picture organization of an input 
pre-compressed bit stream, e.g., by scanning the bit 
stream by about one GOP, this results in processing 
delays and the need for additional memory space, which 
increases costs. Moreover, a further complication is 

15 that the extra memory requirement will vary for programs 
with different GOP lengths. 

The present invention operates on the assumption- 
that a priori knowledge of program GOP structures is not 
available to the transcoders. The invention starts with 

2 0 a reasonable assumed program picture organizations (GOP 

length and sub_GOP length) , and provides gradual 
adjustments to the assumed organization when necessary. 
A GOP length of fifteen pictures, and a sub GOP length 
of three pictures is a reasonable assumed organization 
25 in many cases. For example, N=15 and M=3 corresponds to 
a picture organization of ...BBIBBPBBPBBPBBPBBI... , while 
N=10 and M=2 corresponds to a picture organization of 
...BIBPBPBPBPBI... 

2. DYNAMIC BIT ALLOCATION ■ 

3 0 A hierarchical dynamic bit allocation strategy for 
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a stat mux system is discussed in U.S. Patent 6,167,084 
to L. Wang and A. Luthra, issued December 26, 2 000, and 
entitled "Dynamic Bit Allocation For Statistical 
Multiplexing Of Compressed And Uncompressed Digital 
5 Video Signals." At the top level of the hierarchy, the 
concept of a super GOP (Groups of Pictures) is 
introduced. Specifically, the programs are conceptually 
divided into super GOPs, each having the same number of 
I, P and B pictures, that are assigned the same nominal 

10 number of bits. 

A super frame is defined at a middle level of the 
hierarchy, which is a collection of frames, one from 
each of the programs at the same frame instant. A super 
frame is assigned a target rate according to its 

15 relative complexity measures. The same idea is extended 
to the regular, individual frame, at a lower level of 
the hierarchy, where a target rate for a frame is 
proportional to its complexity measure. 

Moreover, to ensure the encoder and decoder buffers 

2 0 never overflow or underflow, and to limit each 

individual average rate, additional constraints are 
applied on the target number of bits for the super frame 
as well as for the regular frame. 

The present invention extends the hierarchical 

2 5 dynamic bit allocation strategy to a stat mux/remux 

system. 

2.1 Super GOP and Nominal Rate 

As shown in FIG. 2, we conceptually divide L input 
video programs into identical super GOPs (L, N) in terms 

3 0 of the number of frames of each picture type. A first 
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super GOP 2 00 includes data frames from program 1 (210) , 
program 2 (220) , ... , through program L (290) . Each 
program segment, e.g., 210, 220, 290, contains a number 
of complete frames from one or more GOPs. A second 
5 super GOP 2 02 includes data frames from program 1 (212) , 
program 2 (222) , ... , through program L (292) . A third 
super GOP 2 04 includes data frames from program 1 (214) , 
program 2 (224) , ... , through program L (294) . 

Here, L is the number of programs, which is given, 
10 and N is the length of the super GOPs, which is set 

equal to the Least Common Multiple (LCM) of the program 
GOP lengths of N n I = 1,2,..., Z , i.e., 

N = LCM(N l9 N 29 ... 9 N L ) m (1) 

For example, if there are two different GOP lengths 
15 for N programs, say nine and fifteen, the super GOP 

length N=45 . Since N is the smallest integer that can 
be divided by all the program GOP lengths, N n I = 1,2,..., Z, , 
the super GOPs(I/,iV) are the smallest identical groups 
containing the same number of frames of each picture 
20 type. A super GOP (L,N) already contains the integer 
number of GOPs for each program 1, that is, N/Nj . 
Super GOPs (L, N) with N defined in (1) contain the same 
number of I, P and B pictures, and hence, they are 
assigned the same nominal number of bits, 
25 T LxN = (LxN)'R channelibpf) (2) 

where R ch annei{bpf) ^ s the average number of bits per frame. 

The number of 1 , P and B pictures in a GOP for 
program 1 are, respectively, N JrI =l, N J# p =N J /M J -N iI =N J /M J -l , 
and N^N.-N^-N.^N.-N^p-l. 
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2.2 Super Frame and Target Rate 

FIG. 3 illustrates a super frame, which is a 
collection of L frames, one from each of L programs at 
the same time instant. 

At each frame instant n, we can further imagine a 
super frame 3 00, which is a collection of L frames or 
pictures, one from each of the L programs taken at the 
same frame instant. For example, frame 310 from program 
1, frame 32 0 from program 2, . . . , through frame 33 0 
of program L are conceptually arranged in the super 
frame 300. Clearly, a super GOP consists of N super 
frames. Moreover, since these L programs may have 
different GOP structures, the L frames in a super frame 
can have different picture types. 
!5 The target number of bits for super frame n is 

given as : 

L 

T n =- <=■ R (3) 

TX n u a ArrCv + *i.P<xArpCi,P + n^JS^C^ 

where C Unt is the complexity measure for frame n of 
program 1 with picture type t <={l,P,B} , and it can be 

20 either C u , C hP or C lB , depending upon the associated 
picture type of I, P or B. TM5 defines a complexity 
measure for a frame as the product of the number of bits 
generated from the frame and the average quantization 
parameter used for the frame. Other complexity measures 

25 are also allowed. 

a { is a constant factor for addressing the spatial 
resolution of program 2. 
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JSj is the quality (or priority) weighting factor 
for program 1, which will be determined by the program 
provider . 

y lnt is a constant factor used to compensate for 

5 the picture type t s{l 9 P 9 S} of frame n of program 2. It 
can be either y } , y p or y Bt depending upon the picture 
type of I, P or B. 

n m n i p an d n i,B are t ^ rie ^remaining number of I, P and 
B pictures, respectively, for program 2 in the current 
10 super GOP. For a new super GOP , they are reset, and 
then decreased by one after a picture of the 
corresponding type is processed. 

R is the remaining number of bits for the current 

L 

super GOP, defined asi? = i? + ^(r /H _, -i?,,^) . Here, T ln ^ and 

i=i 

15 are, respectively, the target and actual rates for 

L 

frame n-1 of program 2. Hence, ^(Tl^i i s the 

/=i 

number of leftover bits from frames n-1 of program 1 = 
1,2,..., I/, which can be either positive or negative. At 
the beginning of a new super GOP, R needs to be updated 
20 as R=R + T LxN , where R on the right side of the equation 
is the number of bits leftover from the previous super 
GOP, which can be positive or negative. 

Note that the numerator on the right side of 
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equation (3) is the sum of complexity measures for all 
the frames in super frame n. On the other hand, the 
denominator can be considered as the sum of complexity 
measures for all the remaining frames in the current 
5 super GOP . Hence, equation (3) actually assigns a 

target number of bits for a super frame in proportion to 
the super frame's complexity measure. 

2.3. Preventing Encoder Buffer Overflow or 
Underflow 

10 To prevent the encoder buffer 17 0 from overflowing 

or underf lowing, certain constraints are applied by 
setting upper and lower bounds for the target rate for 
super frames. This can be achieved according to the 
techniques set forth in the aforementioned U.S. Patent 

15 No. 6,167,084. 

2.4 Target Rate for Regular Frame 

Given a target number of bits for a super frame n, 
T n , the target number of bits for frame n of program 1 
within the super frame, T hl , can be calculated as: 

20 T i.n=~ T m . (4) 

/=1 

Here, the numerator on the right is the complexity 
measure for frame n of program 1, and the denominator 
can be considered as the complexity measure for super 
frame n. Hence, a frame is assigned a target rate 
25 proportional to its complexity measure. 

The target rates determined by equation (4) are 
based upon the frame relative complexity measure. For 
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10 



15 



20 



transcoders, they may need further adjustment. 
Specifically, the target rate for a frame determined by 
(4) could be larger, or smaller, than the old rate for 
the frame in the input pre-compressed bit stream. A 
rate- conversion transcoder is, however, a device for 
converting a pre-compressed video bit stream into 
another bit stream at a new rate. It cannot improve the 
quality of a pre-compressed video signal regardless of 
whether the new bit rate is smaller or greater than the 
old bit rate. 

Thus, allocating more bits for a (pre-compressed) 
frame wastes the bits. Hence, in accordance with the 
invention, if the target rate for frame n of programs I, 
T ln , (4) is greater than the old rate for the frame in 

the input pre-compressed bit stream, say R oldJt „ , the old 
rate should be applied, i.e., 



This can be achieved using the example bit rate 
functions 144, 154 of FIG. 1 by measuring the bit rate 
of a picture in the received bit stream, e.g., using a 
bit counter, and sending this information to the rate 
control processor 110 to reduce the allocated bits, T, 
for the associated transcoder, if required. 

2.5 Preventing Decoder Buffer Overflow or Underflow 
FIG. 4 illustrates a decoder for receiving data 
from the stat mux/remux of FIG. 1, where packets of a 
selected program are extracted and decoded. At the 
receiving end 480, the decoder 495 is allowed to select 
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the desired program and extract (de-multiplex) the 
corresponding packets via a demux 485. The de- 
multiplexed bit stream is at a variable rate. It is the 
encoder's responsibility to ensure the decoder buffer 
5 490 never overflows or underflows. Additional 

constraints are therefore applied at the encoder. 

This can be achieved using the techniques of the 
aforementioned U.S. Patent No. 6,167,084 to apply a 
constraint on the target rate for each regular frame . 
10 2.6. Constraint on Max and Min Rate 

We can also control the average bit rate over a 
certain number of frames by limiting the target number 
of bits for each frame within a specific range. This 
can be achieved using the techniques of the 
15 aforementioned U.S. Patent No. 6,167,084. 

3 . CHANGES IN PROGRAM GOPS 

The program GOP structure (including GOP length, 
type and arrangement of pictures, and sub GOP length) 
plays an important role in determining a target rate for 

20 a frame, as shown in equations (3,4) , However, a 

program GOP structure may vary from time to time. For 
example, for a transcoder, the input video bit stream 
may consist of segments of different materials/content, 
such as films, news, sports, music, etc., that are pre- 

25 encoded with different GOP structures. Another example 
is commercial insertion, where the inserted commercial 
may be pre-coded with a different GOP structure than the 
original program to be inserted. 

In accordance with the invention, the bit 
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allocation adapts to changes in the program GOP 
structures, e.g., by adjusting the size of the super GOP 
and other parameters for determining the target rates of 
frames . 

5 3.1 Program sub_GOP Length 

The following shows a general form of picture 
organization for an example program 1. The first row is 
in display order and the second row is the corresponding 
encoding order : 

•••^»i^;n-(Af ; -I)^ffi-(A/ / -l)+l "'^ m-l^m+M, &m+l '"^ m+M I -l^ > m+2M l '"^m+Nj * 

Here, the subscripts are the picture's temporal 
reference, and N t and M, are the GOP length and sub_GOP 
length, respectively. The pictures in the compressed 
15 video bit stream that are input to a transcoder are in 
encoding order. The display order is recovered at the 
decoder . 

For an encoder that encodes uncompressed video 
data, the changes in sub_GOP length of a program are 
2 0 available since the GOP program structure is available, 
thus allowing the encoder to plan ahead for bit 
allocation. 

However, for a transcoder, the changes in sub_GOP 
structures are embedded in the input pre-compressed bit 
25 stream. In accordance with the invention, the sub_GOP 
length of a program is calculated at each current 
frame/picture (I, P or B) if the previous frame is an I 
or a P picture by checking the temporal references of 
the current picture and the previous I or P picture. If 
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the previous frame is a B-picture, we do not check the 
sub_GOP. In MPEG syntax, the temporal reference of a 
picture is included in the picture header that is 
located at the beginning of the segment of compressed 
5 bits for the picture. Specifically, the term 

11 temporal_ref erence" is a 10 -bit unsigned integer 
associated with each input picture. It is incremented 
by one, modulo 1024, for each input frame. For example, 
temporal_reference may have values of 67, 68, 69 and so 

10 forth. Hence, a transcoder (or, more specifically, the 
associated GOP structure monitors 142 or 152 of FIG. 1) 
can access the temporal reference of a current picture 
before the transcoder actually processes (transcodes) 
the picture. The temporal reference of the previous I 

15 or P picture is already available because the picture 
has been processed. 

FIG. 5 illustrates a video program with a change in 
the sub_GOP length, and subsequent adjustment of the 
super GOP length, in accordance with the present 

20 invention. 

Note that the program GOPs for the different 
programs 1,..., L do not necessarily align with one 
another . 

An example program 1 that is transcoded includes 
25 GOPs 546, 548 and 550 having a first GOP structure (GOP 
length and/or sub_GOP length), and GOPs 552, 554 and 556 
having a second, different GOP structure. Note that the 
GOP structure can change for more than one program at 
the same time. The super GOP length and bit budget are 
3 0 adjusted accordingly. 
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Moreover, as mentioned, the programs may comprise 
transcoded data only, or both transcoded and encoded 
data . 

At a time indicated at 510, an old super GOP 53 0 
5 with length, N old , is determined. A sub_GOP (the 

distance between two P-pictures in either encoding or 

display order) for each program does not change up until 

a time indicated at 520, when a new super GOP length, 

N new , is determined. Let temp _ref _pic\ and temp __ref _ pic2 
10 be -the temporal references for the previous I or P 

picture and the current picture for program 1, 

respectively. 

If the previous picture is an I or P picture, and 

the current picture is a B picture, the B picture should 
15 be displayed before the I or P (in display order) . 

Thus, the B picture has a smaller temporal reference. 

Now, if the current picture is an I or P picture, the 

actual sub__G0P length for program I is : 

M factual = tem P _ re f _ P ic2 " tem P - re f - P icl ■ ( 6 ) 

2 0 Otherwise, if the current picture is a B picture, 

M factual = tem P - re f - P icl - tem P _ re f - P ic2 + 1 • ( 7 ) 

No sub_GOP is determined for a current picture 
whose previous picture is a B-picture. The start of the 
current picture in FIG. 5, which is an I picture and the 
25 first picture of the GOP 552, is at a time 515. The 
start of the next picture is at a time 52 0. 

Clearly, if M l - M lactual , the current sub__GOP for 
program 2 is correct. Hence, no further action is 
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needed. However, if M , * M l%ectuag , we set M , = M lacttta{ , and 
adjust other parameters as follows: 

1. First, the program GOP length, N t , has to be 
divisible by the new M { (i.e., integrally divisible - 

5 with no remainder) . If not, we adjust it by 

N l ^[N l IM i yM } ( 8 ) 
where [/] denotes integer division with truncation of 
the result toward zero. 

For example, assume N 1= =15 and M=3 initially. Then, 
10 M changes to M=2 . Using eqn. 8, the adjusted program 
GOP length N x =14 . 

2. Second, if N } changes, we reset the super GOP 
length by using equation (1) . The new super GOP 54 0 now 
starts right at the following picture, at time 520. 

15 Note that the program GOPs do not have to be aligned 
with the new super GOPs . The number of I, P and B 
pictures in the (new) super GOP 54 0 can also be 
calculated with the new super GOP length, say N new , as, 
respectively, n J(I =N 1#I * (N/N,) =N/N J# n J#p =N IrP * (N/N 2 ) =N/M I - 

20 n ltl , and n J#B =N I<B * (N/N,) ^N-n^p-n^ . 

3 . Finally, the nominal number of bits for the 
(new) super GOP is now equal to: 

^Z.x# nci „ = (L X N new ) ' R c i m nnel{bpf) • ( 9 ) 

The number of bits for the first (new) super GOP 
25 540, however, may be slightly different than for other, 
subsequent super GOPs that follow the super GOP 54 0 
since the previous super GOP 53 0 (with length N old ) is an 
unfinished super GOP that has leftover bits that were 
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previously allocated, but not yet used. In accordance 
with the invention, these bits can be allocated to the 
first new super GOP 540. Specifically, let N old be the 
old super GOP length and N' be the number of frames of 
5 program I in the old super GOP that have been processed. 
The remaining number of bits for the first new super GOP 
is then: 

R = t LxNmw + [ T L „ N , - (T LxNm - *)] . (10) 
Here, T LxN , is the nominal number of bits for LxN' 
10 frames and (T^ —R) is the number of bits used for the 
frames in the old (unfinished) super GOP 530. Hence, 
ITlxn* ~ (? LxNm — R)] denotes the leftover bits from the old 

(unfinished) super GOP 53 0. 

With the updated information, we can determine the 

15 target rates for the current and future frames (eqns. 3 
and 4) . Note that the sub_G0P length of a program can 
be verified and/or corrected at each frame if the 
previous frame is an I or a P picture, and if necessary, 
other parameters are also adjusted accordingly (eqns. 8- 

20 10) . 

3 . 2 Program GOP Length 

Let N lold and N lnew be the GOP lengths of the 

previous and the current GOPs for program I, 
respectively. There are three possible cases regarding 
2 5 the GOP lengths, i.e., 

M,, tt „ v > N Lold (11) 
N hnew < N lMd 
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For encoders, both N lold and N lnew are available. 
However, for a transcoder, the information on GOP length 
is embedded in the input pre-compressed bit stream. 
Before completely processing the current GOP of program 
5 N lnew is not available. 

We set the current GOP length N ( = N LoId for the 
transcoder. Clearly, if N lnw = N l = N lold , we have a 

correct GOP length for program I. Hence, no further 
action is needed. 
10 FIG. 6 shows an example where the new GOP length 

for the current GOP is longer than the currently used 
GOP length. 

Program 1 includes GOPs 646, 648 and 652 having a 
first GOP structure, and GOPs 654, 656 and 658 having a 
15 second, different GOP structure. 

An old super GOP 630 starts at a time 610, while a 
new super GOP 640 starts at a time 620. Old GOPs 
extend, e.g., from a time 632 to a time 635, and from a 
time 635 to a time 620. A new GOP extends from a time 
20 635 to a time 650. 

If N ln(nv > N; = N lold t i.e., the actual (new) program 
GOP length is longer than the currently used (old) GOP 
length. The GOP structure monitor (142, 152) expects to 
reach a I picture at the end of the current GOP of N i 
25 frames, at time 620. However, the end of a program GOP 
is not actually reached at time 620. In accordance with 
the invention, this alerts the GOP structure monitor of 
a change in the GOP structure. Specifically, it can be 
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concluded that the actual GOP length is longer than the 
currently-assumed GOP length. 

Moreover, because of the nature of the program GOP 
structure (FIG. 2) , the new GOP length will be at least 
5 M t (one sub GOP length) additional pictures longer. 

Here, we have assumed that the sub-GOP length M does 
not change within a GOP. We extend the current GOP of 
N t pictures by M t additional pictures, i.e., 

N^fy+M;. (12) 

10 With the new N t , we recalculate the super GOP 

length by equation (1) , and the nominal number of bits 
for the first new super GOP (and the following super 
GOPs) by equation (2) . The first new super GOP 64 0 is 
aligned with the current P picture of program 1, as 

15 shown at time 620. Because of the leftover bits from 
the previous super GOP 63 0, the number of bits for the 
first new super GOP 640 is re-allocated as: 

* = T L«N new HTw -(^kau -*)] • (13 ) 

where N M and N new are the old and new super GOP 

2 0 lengths, respectively, and N 1 is the number of frames 
of program 1 in the old super GOP 63 0 that have been 
processed. If the extended GOP for program 1 is still 
shorter than the actual GOP, we repeat the above 
procedure (e.g., increasing N x by eqn. 12) until 

25 reaching the end of the actual GOP. 

FIG. 7 shows an example where the new GOP length 
for the current GOP is shorter than the currently used 
GOP length. 

f 
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Program 2 includes GOPs 74 6, 74 8 and 752 having a 
first GOP structure, and GOPs 754, 756, 758 and 760 
having a second, different GOP structure. 

An old super GOP 73 0 starts at a time 710, while a 
new super GOP 740 starts at a time 720. An old GOP 
extends from a time 732 to a time 73 5, and from a time 
735 to a time 750. A new GOP extends from a time 735 to 
a time 72 0. 

If N iw < N i = Ni,oid > i.e., the actual GOP length is 
shorter than the currently used GOP length, an I picture 
(which denotes the start of a new GOP - there is only 
one I picture in a GOP) will be reached (at time 720) 
before the end of the current GOP of N t frames . At 
this point, the GOP structure monitor knows the actual 
GOP length, N lnew , since the GOP 754 has just concluded. 

We set N l ^N lnew . Moreover, with the new program GOP 
length, N l , we reset the super GOP length by (1), and 
recalculate the nominal number of bits for the new super 
GOPs by (2) . Again, because of the leftover bits from 
the previous (unfinished) super GOP 730, the number of 
bits for the first new super GOP 740 may be slightly 
different than for subsequent super GOPs, that is, 

R = + - (T LxNm - *)] . ( 14 ) 

where N old and N new are the old and new super GOP 
5 lengths, respectively, and N A is the number of frames 
of program 2 in the old super GOP 73 0 that have been 
processed. 

4 . RATE CONTROL 
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Once the target rate for a frame of a program is 
set, the next step is to achieve this rate, e.g., by- 
adjusting the coding parameters, such as the 
quantization parameter, in the encoders and transcoders 
5 of the stat mux/remux system. Any suitable rate control 
scheme, such as those for use with the MPEG encoder, can 
be used. Some of them can be applied to the transcoders 
as well. 

Preferably, joint rate control is used to 
10 dynamically distribute the channel capacity among the 

programs according to the programs 1 relative complexity 
measures . 

5 . CONCLUSION 

A stat remux system is presented having encoders 

15 and transcoders that can handle both uncompressed 
digital video signals and pre-compressed video bit 
streams. The system implements a novel adaptive rate 
control, which dynamically distributes the channel : 
capacity over the input video programs, either 

20 uncompressed digital video signals or pre-compressed 

video bit streams, on a frame -by- frame basis. The bit 
allocation strategy adopted in the adaptive rate control 
is able to address changes in picture organization 
(e.g., changes in GOP length or sub_GOP length) in the 

25 input pre-compressed bit streams. When such a change is 
detected, the bit allocation is recalculated. 

Moreover, hierarchical bit allocation is provided 
on a super GOP level, then on a super frame level, and 
finally at an individual frame level. 

3 0 Although the invention has been described in 
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connection with various specific implementations, it 
should be appreciated that various adaptations and 
modifications can be made thereto without departing from 
the scope of the invention as set forth in the claims. 
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What is claimed is : 

1. A method for coding a plurality L of video 
programs, comprising the steps of: 

allocating bits for coding the video programs 
according to a hierarchical scheme that includes at 
least: (a) a super group of pictures (GOP) level, 
wherein a super GOP comprises at least one GOP from each 
of the video programs, and (b) an individual frame 
level ; 

wherein said allocating step is responsive to 
respective GOP structures of the video programs ; 

monitoring the respective GOP structures of the 
video programs to detect any changes thereof; and 

when a change in the GOP structure of at least one 
of the video programs is detected, adjusting the 
allocation of bits for coding the video programs in the 
super GOP according to said change. 

2. The method of claim 1, wherein: 

said monitoring step monitors the GOP structure of 
the video programs on a frame -by- frame basis. 

3. The method of claim 1, wherein: 

the change in GOP structure is due to a change in a 
video source . 

4. The method of claim 1, wherein: 

the change in GOP structure is due to a commercial 
insertion in the associated video program. 
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5. The method of claim 1, wherein: 

an aggregate of the transmission rates for the 
video programs is maintained within a total available 
channel rate. 

6. The method of claim 1, wherein: 

the video programs are communicated over a 
broadband communication network to a decoder population. 

7. The method of claim 1, wherein: 

the hierarchical scheme includes a super frame 
level between the super GOP level and the frame level; 
and 

each super frame comprises L pictures at a common 
frame instant, one from each of the video programs. 

8. The method of claim 1, wherein: 

the L video programs have respective variable bit 
rates . 

9. The method of claim .1, comprising the further 
step of: 

determining whether a target bit rate that is set 
in said allocating step for a frame of pre-compressed 
data of at least one of the video programs is greater 
than a measured bit rate of the frame and, if so, 
reducing the target bit rate to the measured bit rate. 

10. The method of claim 1, wherein: 
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the plurality of video programs comprise at least 
two pre-compressed video programs, and the coding 
thereof comprises transcoding. 

11. The method of claim 10, wherein: 

the plurality of video programs further comprise at 
least one video program having uncompressed video data. 

12. The method of claim 1, wherein: 

the GOP structure comprises a program GOP length. 

13. The method of claim 12, comprising the further 
step of : 

starting a new super GOP, and allocating bits 
thereto, when a change in the length of a program GOP of 
at least one of the video programs is detected. 

14. The method of claim 12, wherein the program 
GOP length is for pre-compressed video data of at least 
one of the video programs, comprising the further steps 
of : 

providing an assumed value for the program GOP 
length before a measurement of the program GOP length 
can be made; 

measuring the program GOP length when the 
measurement can be made/ and 

comparing the measured program GOP length to said 
value and, if there is a difference, setting said value 
to the measured program GOP length. 
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15. The method of claim 13, wherein: 

the allocation of bits to the new super GOP 
accounts for unused bits from a previous super GOP that 
were previously allocated thereto but not used. 

16. The method of claim 13, comprising the further 
step of : 

calculating a length of the new super GOP based on 
the changed- program GOP length. 

17. The method of claim 15, wherein: 

the unused bits are determined based on a number of 
frames processed in the previous super GOP. 

18. The method of claim 1, wherein: 

the GOP structure comprises a sub GOP length. 

19. The method of claim 18, comprising the further 
step of : 

starting a new super GOP, and allocating bits 
thereto, when a change in the length of a sub GOP of at 
least one of the video programs is detected. 

20. The method of claim 18, wherein: 

the sub GOP length is calculated after bits are 
allocated in said allocating step to an I picture. 

21. The method of claim 18, wherein: 

the sub GOP length is associated with a distance 
between successive P-pictures in a program GOP. 
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22. The method of claim 18, wherein: 

when a current picture of one of the video programs 
is an I or P picture, and a previous picture is an I or 
P picture, the sub GOP length is determined according to 
a difference between temporal references of the current 
picture and the previous picture. 

23. The method of claim 18, wherein: 

when a current picture of one of the video programs 
is a B picture, and a previous picture is an I or P 
picture, the sub GOP length is determined according to a 
difference between temporal references of the current 
picture and the previous picture, plus one picture. 

24. The method of claim 18, comprising the further 
step of: 

calculating the sub GOP length after processing an 
I or P picture. 

25. The method of claim 18, wherein the sub GOP 
length is for pre- compressed video data of at least one 
of the video programs, comprising the further steps of: 

providing an assumed value for the sub GOP length 
before a measurement of the sub GOP length can be made; 

measuring the sub GOP length when the measurement 
can be made; and 

comparing the measured sub GOP length to said value 
and, if there is a difference, setting said value to the 
measured sub GOP length. 
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26. The method of claim 18, wherein: 

when a change in the sub GOP length is detected, 
calculating an associated new program GOP length. 

27. The method of claim 26, wherein: 

the new program GOP length N 1#new is calculated 
according to N 1#nw - [N a , old /M Ifnew ] *M I , new ; 

where N lf0ld is the previous program GOP length; 
Mj,new is the new sub GOP length; and 

"/" denotes integer division with truncation of the 
result toward zero. 

2 8 . The method of claim 26, wherein: 
after the new program GOP length is calculated, 
calculating a new super GOP length in accordance 
therewith, and allocating bits to the new super GOP 
accordingly. 

29. The method of claim 1, wherein: 

the GOP structure denotes picture types and their 
arrangement in a program GOP. 

30. The method of claim 1, wherein: 

a length of the super GOP is determined according 
to a least common multiple of the program GOP lengths of 
the respective video programs. 

31. An apparatus for coding a plurality L of video 
programs , compr i s ing : 
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means for allocating bits for coding the video 
programs according to a hierarchical scheme that 
includes at least: (a) a super group of pictures (GOP) 
level, wherein a super GOP comprises at least one GOP 
from each of the video programs, and (b) an individual 
frame level; 

wherein said allocating means is responsive to 
respective GOP structures of the video. programs ; 

means ' for monitoring the respective GOP structures 
of the video programs to detect any changes thereof; and 

means for adjusting the allocation of bits for 
coding the video programs in the super GOP, when a 
change in the GOP structure of at least one of the video 
programs is detected, according to said change. 

32. An apparatus for coding a plurality L of video 
programs , comprising : 

at least two transcoders for receiving pre- 
compressed data of corresponding ones of the video 
programs ; 

a rate control processor for allocating bits for 
coding the video programs according to a hierarchical 
scheme that includes at least: (a) a super group of 
pictures (GOP) level, wherein a super GOP comprises at 
least one GOP from each of the video programs, and (b) 
an individual frame level; 

wherein said rate control processor is responsive 
to respective GOP structures of the video programs; and 

a GOP structure monitor associated with each of the 
transcoders for monitoring the respective GOP structures 
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of the associated video programs to detect any changes 
thereof ; 

> 

wherein said rate control processor adjusts the 
allocation of bits for coding the video programs in the 
super GOP, when a change in the GOP structure of at 
least one of the video programs is detected, according 
to said change. 
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